Macro inspection systems, apparatus and methods

ABSTRACT

The disclosed technology relates to an inspection apparatus that includes a stage configured to retain a specimen for inspection, an imaging device having a field of view encompassing at least a portion of the stage to view a specimen retained on the stage, and a plurality of lights disposed on a moveable platform. The inspection apparatus can further include a control module coupled to the imaging device, each of the lights and the moveable platform. The control module is configured to perform operations including: receiving image data from the imaging device, where the image data indicates an illumination landscape of light incident on the specimen; and automatically modifying, based on the image data, an elevation of the moveable platform or an intensity of one or more of the lights to adjust the illumination landscape. Methods and machine-readable media are also contemplated.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/738,022, filed Jan. 9, 2020, which is a continuation of U.S. patentapplication Ser. No. 16/262,017, filed Jan. 30, 2019, now U.S. Pat. No.10,545,096, issued Jan. 28, 2020, which are incorporated by reference intheir entireties.

TECHNICAL FIELD

The present disclosure generally relates to macro inspection systems,apparatus and methods. More particularly, embodiments of the presentinvention relate to macro inspection apparatus having multiple modes ofillumination that can each provide variable illumination landscapes todetect features on a specimen.

BACKGROUND OF THE INVENTION

Microscopic examination of a specimen to detect specimen features can belimited. Specimens as understood by a person of ordinary skill in theart refer to an article of examination (e.g., a wafer or a biologicalslide), and features refer to known characteristics of a specimen, aswell as abnormalities and/or defects. Features can include but are notlimited to: circuits, circuit board components, biological cells,tissue, defects (e.g., scratches, dust, fingerprints). In some cases,features of a specimen are distributed across relatively large surfaceareas, or a specimen itself is rather large. For such specimens,microscopic examination can be insufficient or undesirable, because suchexamination acquires information over relatively small surface areas andrequires the capture of multiple images of discrete portions of aspecimen in order to represent the entire specimen. In addition,microscopic examination can be limited in the type and variety ofillumination it can provide. For purposes of this specification,microscopic refers to an area less than 0.5 cm².

Accordingly, it is desirable to provide a new mechanism for macroscopicexamination of a specimen that can capture the entire or large areas ofa specimen in a single field of view and can provide for multiple modesof illumination including, but not limited to brightfield, darkfield oroblique illumination; polarized light; cross-polarized light; anddifferential interference contrast (DIC), phase contrast. It is alsodesirable that each mode of illumination provide variable illuminationlandscapes, as explained herein, to detect features of a specimen. Forpurposes of this specification, macroscopic refers to an area 0.5 cm² orgreater.

SUMMARY OF INVENTION

In some aspects, the disclosed technology relates to an inspectionapparatus that includes a stage configured to retain a specimen forinspection, an imaging device having a field of view encompassing atleast a portion of the stage to view light reflected by a specimenretained on the stage, and a plurality of lights disposed on a moveableplatform. The inspection apparatus can further include a control modulecoupled to the imaging device, each of the lights, and the moveableplatform, wherein the control module is configured to perform operationscomprising: receiving image data from the imaging device, wherein theimage data indicates an illumination landscape of light incident on thespecimen, and automatically modifying an elevation of the moveableplatform or an intensity of one or more of the plurality of lights toadjust the illumination landscape, based on the image data.

In another aspect, the control module is further configured to performoperations comprising automatically adjusting a color of one or more ofthe plurality of lights to improve the illumination landscape, based onthe image data.

In some aspects, the control module is further configured to performoperations comprising automatically adjusting a pivot of one or more ofthe plurality of lights to adjust the illumination landscape, based onthe image data.

In some aspects, the control module is further configured to performoperations comprising receiving specimen data indicating a specimenclassification associated with the specimen, and wherein automaticallymodifying an elevation of the moveable platform or an intensity of oneor more of the plurality of lights is further based on the specimendata.

In some aspects, the control module is further configured to performoperations comprising receiving specimen data indicating one or morephysical properties associated with the specimen, and whereinautomatically modifying an elevation of the moveable platform or anintensity of one or more of the plurality of lights is further based onthe specimen data.

In some aspects, the control module is further configured to performoperations comprising automatically adjust, based on the specimenclassification, focus of a lens used for viewing the specimen.

In some aspects, the plurality of lights are positioned radially outsidea perimeter of the stage.

In one aspect, a computer-implemented method for automatically adjustingan illumination landscape provided by an inspection apparatus comprisesreceiving, at a control module coupled to an imaging device of theinspection apparatus, image data from the imaging device, wherein theimage data indicates the illumination landscape of light incident on aspecimen; analyzing, at the control module, the image data to determinea specimen classification corresponding to the specimen retained on astage of the inspection apparatus; and automatically modifying, based onthe specimen classification and by the control module, an elevation of amoveable platform of the inspection apparatus or an intensity of one ormore of a plurality of lights disposed on the moveable platform toadjust the illumination landscape.

In some aspects, the method further comprises automatically adjusting,based on the specimen classification, a color of the one or more of theplurality of lights to adjust the illumination landscape.

In some aspects, the method further comprises automatically adjusting,based on the specimen classification, a pivot of the one or more of theplurality of lights to adjust the illumination landscape.

In some aspects, the method further comprises automatically adjusting,based on the specimen classification, focus of a lens used for viewingthe specimen.

In some aspects, analyzing the image data to determine the specimenclassification further comprises identifying, from the image data, afeature of the specimen and using the feature to determine the specimenclassification.

In some aspects, the method further comprises referencing a profiledatabase using the specimen classification and obtaining, from theprofile database, an illumination profile, wherein the illuminationprofile is associated with the specimen classification in the profiledatabase.

In one aspect, a non-transitory computer-readable storage medium storesthereon executable instructions that, as a result of being executed byone or more processors of a control module coupled to an imaging deviceof an inspection apparatus, cause the control module to obtain imagedata from the imaging device, wherein the image data indicates anillumination landscape of light incident on a specimen; analyze theimage data to determine a specimen classification corresponding to thespecimen retained on a stage of the inspection apparatus; andautomatically modify, based on the specimen classification, an elevationof a moveable platform of the inspection apparatus or an intensity ofone or more of a plurality of lights disposed on the moveable platformto adjust the illumination landscape.

In some aspects, the executable instructions further cause the controlmodule to adjust, based on the specimen classification, a color of theone or more of the plurality of lights to adjust the illuminationlandscape.

In some aspects, the executable instructions further because the controlmodule to adjust, based on the specimen classification, a pivot of theone or more of the plurality of lights to adjust the illuminationlandscape.

In some aspects, the executable instructions further cause the controlmodule to query a profile database using the specimen classification;obtain, from the profile database, an illumination profile, wherein theillumination profile is associated with the specimen classification inthe profile database; and use the illumination profile to determine anadjustment to adjust the illumination landscape.

In some aspects, the executable instructions that cause the controlmodule to analyze the image data to determine the specimenclassification further cause the control module to identify, from theimage data, one or more physical properties associated with thespecimen; and determine, based on the one or more physical properties,the specimen classification.

In some aspects, the executable instructions further cause the controlmodule to adjust, based on the specimen classification, focus of a lensused for viewing the specimen.

In some aspects, the executable instructions that cause the controlmodule to analyze the image data to determine the specimenclassification further cause the control module to use the image data asinput to a machine learning classification model, wherein an output ofthe machine learning classification model is the specimenclassification.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features of the disclosure can be obtained, a moreparticular description of the principles briefly described above will berendered by reference to specific embodiments thereof which areillustrated in the appended drawings. Understanding that these drawingsdepict only exemplary embodiments of the disclosure and are nottherefore to be considered to be limiting in their scope, the principlesherein are described and explained with additional specificity anddetail through the use of the accompanying drawings in which:

FIG. 1A shows a perspective view of a particular embodiment of a macroinspection apparatus;

FIG. 1B shows a perspective view of a particular embodiment of a macroinspection apparatus;

FIG. 1C shows a particular embodiment of a macro inspection apparatus;

FIG. 2A shows a schematic side view of an embodiment of a macroinspection apparatus providing oblique illumination from a low angle ofincidence;

FIG. 2B shows an example of the low angle oblique illumination asestablished by the operation of the apparatus per FIG. 2A;

FIG. 3A shows a schematic side view of an embodiment of a macroinspection apparatus providing oblique illumination from a high angle ofincidence;

FIG. 3B shows an example of high angle oblique illumination asestablished by the operation of the apparatus per FIG. 3A;

FIG. 4A shows a schematic view of a macro inspection apparatus operatingin a darkfield illumination mode;

FIG. 4B shows an example of darkfield illumination as established by theoperation of the apparatus per FIG. 4A;

FIG. 5A shows a front view of an exemplary light bar;

FIGS. 5B and 5C show side views of the light bar of FIG. 6A, showing anability to pivot as at angles α1 and α2;

FIG. 6 shows a schematic top plan view of an embodiment of a light ringassembly employing multiple light bars for establishing variouspotential illumination vectors;

FIG. 7A shows a schematic side view of a macro inspection apparatusemploying two opposed brightfield lights positioned to illuminate aspecimen in a substantially orthogonal direction;

FIG. 7B shows an example of substantially orthogonal illumination asestablished by the operation of the apparatus per FIG. 7A;

FIG. 8A shows an example implementation of first and second arrays ofbrightfield lights;

FIG. 8B shows an example of substantially orthogonal illuminationemploying 2 lights;

FIG. 8C shows an example of substantially orthogonal illuminationemploying 7 lights;

FIG. 9A shows a schematic side view of a macro inspection apparatusemploying orthogonal illumination through the lens;

FIG. 9B shows an example of orthogonal illumination as established bythe operation of the apparatus per FIG. 9A;

FIG. 10 shows the general configuration of an embodiments of a computeranalysis system.

FIG. 11 shows example method steps for calibrating a macro inspectionsystem to achieve different illumination landscapes;

FIG. 12 shows an example coordinate system to define an area ofillumination projected by the lights of a macro inspection system;

FIG. 13A shows example method steps for illuminating a specimen using amacro inspection system;

FIG. 13B illustrates steps of an example process for identifying aspecimen classification and automatically adjusting an illuminationlandscape of the macro inspection apparatus;

FIG. 14 shows example shows an example training model that uses certaininputs and outputs to feed into an artificial intelligence algorithm togenerate one or more illumination profiles;

FIG. 15 shows an example coordinate system for a rectangular specimen;

FIG. 16A shows an irregularly shaped specimen;

FIG. 16B shows an example coordinate system for an irregularly shapedspecimen; and

FIG. 17 shows a method for establishing an angle of rotation of afeature on a specimen relative to a coordinate system.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

In accordance with some embodiments of the disclosed subject matter,mechanisms (which can include systems, methods, devices, apparatuses,etc.) for macroscopic inspection of specimens are provided. Macroscopicexamination (sometimes referred to as inspection) refers to scanning,imaging, analyzing, measuring and any other suitable review of aspecimen using the disclosed macroscopic inspection mechanism. Thedisclosed macroscopic inspection mechanism includes one or more modes ofillumination that can each provide variable illumination landscapes, asdescribed herein. Although the following description refers tocomponents and methods implemented in a macroscopic inspectionmechanism, the components and methods described herein can also beimplemented in a microscope inspection system.

FIGS. 1A, 1B and 1C illustrate examples of a macro inspection system 10according to some embodiments of the disclosed subject matter. At a highlevel, the basic components of macro inspection system 10, according tosome embodiments, include an illumination unit (e.g., light assemblyring 26) for providing light to a specimen S, a focusing lens 14, animaging device 24, a specimen stage 12, a control module 70 comprisinghardware, software, and/or firmware and a computer analysis system 75.Macro inspection system 10 can be implemented as part of an opticalinspection system that uses transmitted or reflected light.

In some embodiments, as shown in FIG. 1A, a light assembly ring 26 canbe used as an illumination unit for macro inspection system 10. One ormore light fixtures, e.g., light bars 28, represented by LB₁ to LB_(x),can be mounted to light assembly ring 26. Note, any type of suitablelight fixture can be mounted to light assembly ring 26. As shown in FIG.1A, light assembly ring 26 can be configured so that the individual oneor more light bars 28 mounted to the assembly are positioned radiallyoutside of the perimeter of specimen stage 12. Each light bar 28 caninclude one or more plurality of lights 16 (as shown in FIG. 5A). Macroinspection system 10 can also include more than one light assembly ring26, as shown for example in FIG. 1B.

In some embodiments, light assembly ring 26 can be configured so that itis movable along guiderails 20. Note, that the illumination unit is notlimited to a ring formation, and individual light bars 28 can be mountedin other types of formations to a non-moveable or moveable platform 18as shown, for example in FIGS. 2A, 3A, 7A and 9A. Further, the movementof movable platform 18 to different positions along the height of theguiderails 20 can be controlled manually, or automatically by software,hardware, and/or firmware (e.g., control module 70). Depending on itsheight in relation to specimen stage 12, light assembly ring 26 can beused to provide oblique or darkfield illumination to a specimen whenretained on specimen stage 12. For example, to provide variable anglesof oblique illumination, light assembly ring 26 can be positioned sothat its light can be projected at different heights above a specimenplane (i.e., the top planar surface of a specimen when positioned onspecimen stage 12). In some embodiments, the specimen plane correspondswith a focal plane of macro inspection system 10 (i.e., the plane wherethe specimen is in focus). In further examples, to provide darkfieldillumination, light assembly ring 26 can be positioned so that its lightcan be projected at the same, or substantially the same, level of thespecimen plane of a specimen on specimen stage 12 to provide darkfieldillumination to a specimen when retained on specimen stage 12.

As used herein: oblique illumination refers to light projected towardthe specimen at an angle of incidence less than 90 degrees and greaterthan 0 degrees, typically greater than 1 degrees; darkfield illuminationrefers to light projected toward the specimen at an angle of incidenceless than 1 degrees and typically 0 degrees; and brightfieldillumination refers to light projected toward the specimen at an angleof incidence perpendicular (90 degrees) to the plane of the specimen.Brightfield illumination can refer to a light source that providesillumination through lens 14 towards a specimen in an orthogonaldirection (“orthogonal illumination”), as shown for example in FIG. 9A,or to a light source positioned outside lens 14 that projects light in a“substantially orthogonal” direction (“substantially orthogonalillumination”), as shown, for example, in FIGS. 7A and 8A.

As shown in FIGS. 2A and 3A, each light bar 28 can provide obliquelighting at different angles of incidence, from multiple directions, inaccordance with some embodiments of the disclosed subject matter. Forexample, as illustrated in FIG. 2A, each light bar 28 is supported overthe specimen plane of specimen stage 12 at a first position P1 providingfor oblique illumination at a first angle of incidence, while, in FIG.3A, each light bar 28 has been moved upward to position P2 to provideoblique illumination at a second angle of incidence. Moving the lightfixture upward can cause the angle of incidence to increase. Theresultant illumination of stage 12 is shown in FIGS. 2B and 3B. In someembodiments, at each height of moveable platform 18 along guiderails 20,light bar 28 can be selectively pivotable about a pivot point as shownfor example in FIGS. 5B and 5C to create varied angles of illuminationrelative to the specimen plane of a specimen when retained on specimenstage 12. Note, a person of ordinary skill in the art will readilyunderstand that the oblique lighting shown in FIGS. 2A and 3A, can beimplemented with a single or multiple light bars 28.

In some embodiments, as shown in FIG. 4A, light bars 28 can bepositioned so that illumination from light bar 28 is substantiallyparallel to a specimen plane to provide darkfield illumination to aspecimen when retained on specimen stage 12. Substantially parallel isto be understood as having an angle of incidence from −1° to +1°, toallow for imperfections in alignment, but in some embodiments, theillumination will be on plane, i.e., at an angle of incidence of 0°,whereby illumination will be reflected only if there are featuresextending off of a perfectly flat planar surface of a specimen. If aspecimen is perfectly flat and featureless, then it would not reflectany of the substantially parallel illumination to lens 14, and such aspecimen viewed by lens 14 (as shown in FIG. 4B) will not beilluminated. If there are protruding imperfections or other features,then the illumination from light bar 28 will reflect off of suchimperfections and/or features and will be captured by image sensor 22via lens 14. Note, a person of ordinary skill in the art will readilyunderstand that the darkfield illumination shown in FIG. 4A, can beimplemented with a single or multiple light bars 28.

In some embodiments, an XY translation stage can be used for specimenstage 12. Specimen stage 12 can be driven by stepper motor, servermotor, linear motor, piezo motor, and/or any other suitable mechanism,including a manual mechanism. Specimen stage 12 can be configured tomove an object in the X axis and/or Y axis directions under the controlof any suitable controller (e.g., control module 70), in someembodiments. An actuator (not shown but known in the art) can be used tomake coarse focus adjustments of, for example, 0 to 5 mm, 0 to 10 mm, 0to 30 mm, and/or any other suitable range(s) of distances. An actuatorcan also be used in some embodiments to provide fine focus of, forexample, 0 to 50 μm, 0 to 100 μm, 0 to 200 μm, and/or any other suitablerange(s) of distances.

In some embodiments, lens 14 can be supported on a lens stage 15 and canbe positioned at an aperture through the lens stage above specimen stage12. Further, macro inspection system 10 can include a focus mechanismthat adjusts specimen stage 12 in a Z direction towards and away fromlens 14 and/or adjusts lens 14 (e.g., via lens stage 15 along guiderails20) towards and away from specimen stage 12. Movement of specimen stage12 and/or lens 14 can be driven by stepper motor, server motor, linearmotor, piezo motor, and/or any other suitable mechanism, including amanual mechanism. Lens 14 can have different magnification powers,and/or be configured to operate with brightfield, darkfield or obliqueillumination, polarized light, cross-polarized light, differentialinterference contrast (DIC), phase contrast and/or any other suitableform of illumination. The type of lens used for macro inspection system10 can be based on desired characteristics, for example, magnification,field of view, numerical aperture, among others. In some embodiments,lens 14 can be a macro lens that can be used to view a specimen within asingle field of view. Note, the term field of view as understood by aperson of ordinary skill in the art refers to an area of examinationthat is captured at once by an image sensor. Further, a person ofordinary skill in the art will readily understand that the terms fieldof view and image are used interchangeably herein.

The illumination of a specimen on specimen stage 12 reflects up to lens14 mounted to an imaging device 24 (e.g., camera), and imaging device 24can capture images and/or video of a specimen. In some embodiments,camera 24 can be a rotatable camera that includes an image sensor,configured to allow the camera to be aligned to a specimen, a stageand/or a feature on a specimen. The image sensor can be, for example, acharged-coupled device (CCD), a complementary metal-oxide semiconductor(CMOS) image sensor, and/or any other suitable electronic device thatconverts light into one or more electrical signals. Such electricalsignals can be used to form images and/or video of an object. In someembodiments, such electrical signals are transmitted for display on adisplay screen connected to macro inspection system 10. Some examplemethods for rotating a camera that can be used by macro inspectionsystem 10 are described in U.S. Pat. No. 10,048,477 entitled “Camera andObject Alignment to Facilitate Large Area Imaging in Microscopy,” whichis hereby incorporated by reference herein in its entirety. In someembodiments, imaging device 24 can be replaced with an ocular or aneyepiece that is used to view a specimen.

In some embodiments, as shown in FIG. 5A, a light bar comprisesindividual lights 16 organized in two rows. Individual lights 16 can bebased on any type of suitable lighting technology, including but notlimited to: light emitting diode (LED), organic light emitting diode(OLED), fluorescent, fiber optic, gas-plasma, cathode ray tube (CRT),liquid crystal display (LCD), laser, etc. In some embodiments, as shownin FIG. 5A, each light can be individually addressed by its light barnumber and light number, as represented by LB_(i)L_(j). In furtherembodiments, the lights can be divided into sections (e.g., by row,column, quadrant, light bar, and/or any other suitable division) andeach section can be addressable. Software, hardware and/or firmware(e.g., control module 70) can control the activation, intensity and/orcolor of each light or section by its address. Activation refers to theturning on of a light, intensity refers to the rate at which lightenergy is delivered to a unit of surface, and color refers to an RGB(red, green, blue) color value, where each color value is specified asan integer from 0 to 255. Intensity can be determined by light meters,image sensors and/or other suitable intensity measurement devices.Plurality of lights 16 can be comprised of lights that projectmonochromatic, different colors, and/or any combination thereof.

As shown in FIG. 6, in accordance with some embodiments of the disclosedsubject matter, a plurality of light bars 28 (e.g., LB₁ to LB₈ can bepositioned radially outside the perimeter of specimen stage 12 (e.g., inan octagonal shape creating a 360 degree circumference of lights) andcan be selectively activated and illuminable by color and/or intensityto illuminate the specimen from different directions. Software, hardwareand/or firmware (e.g., control module 70) can control which light barsand individual lights are activated and at what color and/or intensity.A single light 16, or multiple lights 16 from a single or multiple lightbars 28 can be activated to illuminate a portion or an entire field ofview at the specimen plane. The type of specimen being examined, thetype of feature being examined, a region of interest on a specimen,and/or any other suitable criteria, can determine which lights areactivated and at what color and/or intensity.

As shown in FIG. 6, according to some embodiments, one or more of lightbars 28 can be connected to a neighboring light bar 28 with a universaljoint 30 providing connectivity between neighboring light bars 28 sothat they can be pivoted concurrently and at the same angle when one ofthe co-joined light bars is moved. In further embodiments, universaljoints 30 can be used at every neighboring junction to allow for commonmovement of all light bars 28 and control by a single control mechanism(e.g., control mechanism 32). Software, hardware and/or firmware (e.g.,control module 70) can control the pivoting of each light bar 28individually or concurrently with one or more other light bars. In someembodiments, light bars 28 can be pivoted manually. Each light bar 28can be pivoted the same or different amounts about a pivot point.

Each individual light 16 of a single light bar 28 (represented byLB_(i)L_(j)) can individually or together emit a vector of light toilluminate a particular area on the specimen plane) “area ofillumination” (The magnitude of this area of illumination can vary fromilluminating a portion of the specimen to encompassing the entirespecimen plane. The area of illumination can be calculated at differentaxial locations above, below or on the specimen plane (e.g., at the topof specimen stage 12, at the top of the specimen plane, at the focalplane, etc.) along the beam of light represented by the vectors. Theareas covered by each vector of light can either be overlapping in partwith the areas covered by the vector of light emitted from a neighboringlight bar or not overlapping at all. In some embodiments, one or morefocusing lenses and/or collimating lenses can be used to focus the areaof each light vector to a region suitable for a specimen on specimenstage 12.

In some embodiments, as shown for example in FIG. 6, multiple light barsare radially positioned around the perimeter of specimen stage 12 andare selectively illuminable to illuminate a specimen from differentdirections. Each light bar, according to some embodiments, can emit asingle vector of light. FIG. 6 shows three illumination vectors of lightemitted from light bars LB₃, LB₄ and LB₅ respectively. The size of thevector can vary, but each vector illuminates from a discrete directionof limited degrees (or minutes of arc) that is less than the entire 360degree circumference of the radially positioned light bars that canproject light towards a specimen on specimen stage 12. Each vector canilluminate at least a portion of the entire field of view, and in someembodiments, each vector can illuminate an entire field of view.

In some embodiments, a single illumination vector ranges from 1 degreeor more to 180 degrees or less (60 or more to 10,800 or less minutes ofarc). In other embodiments, a single illumination vector ranges from 45degrees or more to 120 degrees or less (2,700 or more to 7,200 or lessminutes of arc), in other embodiments, from 30 degrees or more to 45degrees or less (1,800 or more to 2,700 or less minutes of arc), inother embodiments, from 10 degrees or more to 30 degrees or less (600 ormore to 1,800 or less minutes of arc), in other embodiments, from 5degrees or more to 10 degrees or less (300 or more to 600 or lessminutes of arc), and, in other embodiments, from 2 degrees or more to 5degrees or less (120 or more to 300 or less minutes of arc). The vectordepends upon the number and position of activated lights of the at leastone light bar 28 relative to the position of the specimen.

Light bar 28 can vary as to the number of lights 16, as represented bythe number of rows and columns, the size of each individual light, thecone angle of each individual light, the pitch (p) between lights andthe distance between the lights and the area where the light isprojected. In some embodiments, the size of specimen stage 12, thespecifications of lens 14, the size and/or type of specimen beinginspected, and/or the features of a specimen that are being examined,can determine the configuration of lights on light bar 28, including,for example, the arrangement of lights (whether in rows and columns orin other arrangements), the total number of lights, the distance, and/orthe pitch (p).

In some embodiments, as shown in FIG. 7A, macro inspection system 10 canalso include one or more brightfield lights 50 positioned radiallyoutside of lens 14 and directed to illuminate at least a portion ofspecimen S in a substantially orthogonal direction. By “substantiallyorthogonal” it is meant that the source of light is positioned outsideof lens 14 and the projected light beam is perpendicular to the specimenand illuminates a portion or all of the specimen (e.g., as shown in FIG.7B). Brightfield lights 50 can be deployed in various configurationsoutside of the lens 14 (e.g., radially or in a grid pattern). Inparticular embodiments, such as that shown in FIG. 8A, a plurality ofbrightfield lights 50 are arranged in a first array of brightfieldlights 52, wherein each brightfield light of the first array is radiallyequidistant from the center of lens 14. While FIG. 8A provides anembodiment with a second array of brightfield lights 54, wherein eachbrightfield light 50 of second array 54 is radially equidistant from thecenter of lens 14, in some embodiments, a single array or more than twoarrays can also be deployed. The second array in FIG. 8A is at a fartherdistance than the first. The arrays are shown as circular about thecircumference of the lens 14, such that lights of the different arrayscan be selected for illuminating the specimen from varying radialdistances. Each light can be addressed, for example, by its array numberand light number (e.g, R_(x)L_(y)). FIG. 8B shows the resultantillumination of specimen stage 12 when lights R₁L₇ and R₁L₃ areactivated. FIG. 8C shows the resultant illumination of specimen stage 12when lights R₁L₇, R₂L₁₁, R₁L₅, R₂L₇, R₁L₃, R₁L₃, R₂L₃, R₁L₁, R₂L₁₅,R₁L₇, R₂L₁₁ are activated.

As shown in FIG. 9A, in some embodiments, macro inspection system 10includes light source 60 (which can include a single or plurality oflights) for providing selective illumination through lens 14 from avertical illuminator 62 and orthogonally illuminating at least a portionof a specimen S. This illumination is termed “orthogonal” as it isdirectly above the specimen, whereas the brightfield illumination shownin FIGS. 7A and 8A emanate from light sources radially outside of lens14 positioned over specimen S (“substantially orthogonal”). In someembodiments, as shown in FIG. 9B, the orthogonal light can encompass theentire specimen. A person of ordinary skill in the art will readilyunderstand that orthogonal or substantially orthogonal light provided inFIGS. 7A, 8A and 9A can include polarized light, cross-polarization ordifferential interference contrast among other lighting techniques.

Similar to lights 16, each light 50 and/or 60 can be individuallyaddressed. In further embodiments, the lights can be divided intosections (e.g., by array, quadrant, and/or any other suitable division)and each section can be addressable. Software, hardware and/or firmware(e.g., control module 70) can control the activation, intensity and/orcolor of each light or section by its address. Plurality of lights 50and/or 60 can be comprised of lights that project monochromatic,different colors, and/or any combination thereof.

As should be generally appreciated from the examples of illumination inFIGS. 2B, 3B, 4B, 7B, 8B, 8C and 9B, the various embodiments of thepresent invention allow for darkfield illumination, illumination atvariable oblique angles and brightfield illumination (both orthogonaland substantially orthogonal illumination).

In some embodiments, control module 70 includes a controller andcontroller interface, and can control any settings of macro inspectionsystem 10 (e.g., intensity of lights 16, 50 and/or 60, color of lights16, 50 and/or 60, turning on and off one or more lights 16, 50 and/or60, pivoting or other movement of one or more light bars 28, movement ofone or more light ring assemblies 26 (e.g., in a z direction), movementof specimen stage 12 (in x, y, and/or z directions), movement of lens14, recording of image data by image sensor 22/camera 24, rotation ormovement of camera 24, processing of illumination data, processing ofimage data). Control module 70 and applicable computing systems andcomponents described herein can include any suitable hardware (which canexecute software in some embodiments), such as, for example, computers,microprocessors, microcontrollers, application specific integratedcircuits (ASICs), field-programmable gate arrays (FPGAs) and digitalsignal processors (DSPs) (any of which can be referred to as a hardwareprocessor), encoders, circuitry to read encoders, memory devices(including one or more EPROMS, one or more EEPROMs, dynamic randomaccess memory) “DRAM” (static random access memory) “SRAM” (and/or flashmemory), and/or any other suitable hardware elements. In someembodiments, individual components within macro inspection system 10 caninclude their own software, firmware, and/or hardware to control theindividual components and communicate with other components in macroinspection system 10.

In some embodiments, communication between the control module (e.g., thecontroller and controller interface) and the components of macroinspection system 10 can use any suitable communication technologies,such as analog technologies (e.g., relay logic), digital technologies(e.g., RS232, ethernet, or wireless), network technologies (e.g., localarea network (LAN), a wide area network (WAN), the Internet, Bluetoothtechnologies, Near-field communication technologies, Secure RFtechnologies, and/or any other suitable communication technologies.

In some embodiments, operator inputs can be communicated to controlmodule 440 using any suitable input device (e.g., keyboard, mouse,joystick, touch).

In some embodiments, control module 70 controls the activation,intensity and/or color of one or more of the plurality of lights 16, 50and/or 60, as well as the position of lights 16 and/or light bar 28(e.g., by adjusting a light bar's height, or by pivoting a light bar),light 50 and/or 60 (e.g., by adjusting the distance between lights 50and/or 60 and the specimen plane) to provide for variable illuminationlandscapes on a specimen when it is placed on specimen stage 12.Illumination landscape refers to the color and/or intensity of light ona region of interest of a specimen as a result of the activation anddistribution of light from the one or more of the plurality of lights16, 50 and/or 60 that is directed towards a specimen. The illuminationlandscape can affect the image viewed through lens 14 and/or imagescaptured by image sensor 22. Control module 70 can control the intensityof one or more of the plurality of lights 16, 50 and/or 60 to provide adesired illumination landscape on a specimen plane and/or specimen stage12. For example, control module 70 can control the intensity of one ormore of the plurality of lights 16, 50 and/or 60 to provide anillumination landscape of uniform intensity on a specimen plane and/orspecimen stage 12. The type of illumination landscape provided can bedetermined by the specimen type, mechanical and/or physical propertiesof a specimen (e.g., specimen size, specimen reflectivity), a specimenfeature being examined, a particular stage of a manufacturing and/orexamining process, or some other suitable variable, individually or inany combination thereof.

In some embodiments, computer analysis system 75 can be coupled to, orincluded in, macro inspection system 10 in any suitable manner using anysuitable communication technology, such as analog technologies (e.g.,relay logic), digital technologies (e.g., RS232, ethernet, or wireless),network technologies (e.g., local area network (LAN), a wide areanetwork (WAN), the Internet) Bluetooth technologies, Near-fieldcommunication technologies, Secure RF technologies, and/or any othersuitable communication technologies. Computer analysis system 75, andthe modules within computer analysis system 75, can be configured toperform a number of functions described further herein using imagesoutput by macro inspection system 10 and/or stored by computer readablemedia.

Computer analysis system 75 can include any suitable hardware (which canexecute software in some embodiments), such as, for example, computers,microprocessors, microcontrollers, application specific integratedcircuits (ASICs), field-programmable gate arrays (FPGAs), and digitalsignal processors (DSPs) (any of which can be referred to as a hardwareprocessor), encoders, circuitry to read encoders, memory devices(including one or more EPROMS, one or more EEPROMs, dynamic randomaccess memory (“DRAM”), static random access memory (“SRAM”), and/orflash memory), and/or any other suitable hardware elements.

Computer-readable media can be any non-transitory media that can beaccessed by the computer and includes both volatile and nonvolatilemedia, removable and non-removable media. By way of example, and notlimitation, computer readable media can comprise computer storage mediaand communication media. Computer storage media can include volatile andnonvolatile, removable and non-removable media implemented in any methodor technology for storage of information such as computer readableinstructions, data structures, program modules or other data. Computerstorage media includes, but is not limited to, RAM, ROM, EEPROM, flashmemory or other memory technology, CD-ROM, digital video disk (DVD) orother optical disk storage, magnetic cassettes, magnetic tape, magneticdisk storage or other magnetic storage devices, or any other mediumwhich can be used to store the desired information and which can beaccessed by the computer.

FIG. 11 shows at a high level, an example calibration method 1100 forcalibrating macro inspection system to achieve different illuminationlandscapes, in accordance with some embodiments of the disclosed subjectmatter. In some embodiments, calibration method 11 can use macroinspection system 10.

At 1101, control module 70 can initialize macro inspection system 10. Insome embodiments, initialization can include determining theconfiguration of lights 16, 50 and/or 60 of macro inspection system 10(e.g., the total number of lights 16, 50 and/or 60, the address andlocation of each light 16, 50 and/or 60, the total number and locationof light bars 28, the area of projection for each light 16, 50 and/or 60at each possible position (including height and angle) from the lightsource to the region where the light is projected (collectively,“configuration information” (and storing the configuration informationin local or remote memory.

In some embodiments, as shown for example in FIG. 12, a Cartesian XYcoordinate system can be used to define an area of illuminationprojected by each light 16, 50 and/or 60. The area of illumination 1210is measured in relation to coordinate axes 1212A and 1212B that meet atorigin point (O). In some embodiments, the coordinate axes can be a pairof perpendicular lines tangent to a specimen and that extend fromreference indices 1213A and 1213B found on a specimen. Note thatcoordinate axes 1212A and 1212B and origin point O are just examples, anarea of illumination 1210 can be measured from other coordinate axes andorigin point O and/or from another reference point(s). In otherembodiments, an area of illumination 1210 can be located by: its polarcoordinates in relation to an origin point and/or any other suitablelocation. In some embodiments, the configuration information at 1101 canbe used in relation to the coordinate system defined for macroinspection system 10 to calculate the area of illumination 1210projected by each light 16, 50 and/or 60, and stored for use by macroinspection system 10 to illuminate a specimen S, or a portion thereof.

At 1102, a reference specimen with known features and/ormechanical/physical properties (e.g., size, reflectivity) can be placedon specimen stage 12. Different combinations of lights 16, 50 and/or 60can be activated at different colors and/or intensities, at differentpossible distances and angles (collectively, “light position” (from thelight source to the region where the light is projected to determine adesirable illumination landscape for the reference specimen (at 1103).In some embodiments, the desirable illumination landscape can bedetermined based on the quality of images captured by image sensor 22,based on the measured intensity of light reflected off a specimen Sacross each individual pixel or pixel groups of image sensor 22, basedon quality of images displayed on a display screen and/or any othersuitable metric. In some embodiments, the illumination landscape can beadjusted by manually activating different combinations of lights 16, 50and/or 60 at different colors and/or intensities and at differentpossible positions until the desired illumination landscape is achieved.In other embodiments, the illumination landscape can be adjusted byprogramming a set of conditions (e.g., using control module 70 andconfiguration information of 1101) to turn on different combinations oflights 16, 50 and or 60 at different colors and/or intensities and atdifferent light positions until a desired illumination landscape isachieved. When the desired illumination landscape for a referencespecimen is achieved, the address (or other identifying information) ofthe activated lights, the intensity level and color of each selectedlight, as well as position information for each selected light and thedistance between stage 12 and lens 14 (collectively “illuminationprofile” (can be stored (at 1104) by control module 70 for future use.

This process to find and store an appropriate illumination profile canbe repeated for different reference specimens representing differentclassification groups—e.g. by specimen type, by similar mechanicaland/or physical specimen properties (e.g., similar reflectivityproperties, similar size dimensions), by feature type, by manufacturingprocess and/or examination stage, by region of interest and/or any othersuitable classification group. This process can also be repeated for thesame reference specimen to find different illumination profiles that areappropriate for different attributes of the specimen (e.g., asdetermined by a specimen's mechanical or physical properties); differentspecimen features that are being examined; different regions of intereston the specimen and/or the manufacturing/examination process that isbeing examined. In some embodiments, a reference specimen is first putin focus before an illumination profile is calculated. In furtherembodiments, the distance between specimen stage 12 and lens 14 isadjusted to different preset distances and an illumination profile iscalculated for a reference specimen at each preset distance.

In embodiments where a uniform illumination landscape is desired, areflective specimen that exhibits a uniform reflective background, asdetermined by standard measurement of reflectivity, can be used tocalibrate macro inspection system 10 to provide a uniform illuminationlandscape. A background can be considered uniform if the reflectivity(e.g., as measured across each individual pixel or pixel groups of imagesensor 22) does not vary by more than 5% across the entire field of viewof the specimen when viewed on specimen stage 12, and preferably lessthan 2%. In some embodiments, a reference specimen without a uniformreflective background can be used to calibrate macro inspection system10 to provide a uniform illumination landscape. When such a specimen isused, lens 14 can be used to create a uniform reflective background bydefocusing the specimen to blur any foreign objects and surfaceirregularities on the specimen to create a more uniform reflectivebackground. The illumination landscape can be adjusted by activatingdifferent combinations of lights 16, 50 and/or 60 at different colorsand/or intensities and at different possible positions until a uniformillumination landscape is achieved. When a uniform illuminationlandscape is achieved, the address (or other identifying information) ofthe activated lights, the intensity and color level of each selectedlight, as well as light position information for each selected light andthe distance between specimen stage 12 and lens 14 can be stored bycontrol module 70 as an illumination profile that provides uniformillumination for macro inspection system 10, a particular specimen, aspecimen class, a region of interest, a particular stage in themanufacturing or examining process, and/or for any other suitableclassification group.

It should be understood that at least some of the portions ofcalibration method 1100 described herein can be performed in any orderor sequence not limited to the order and sequence shown in and describedin connection with FIG. 11, in some embodiments. Also, some portions ofprocess 1100 described herein can be performed substantiallysimultaneously where appropriate or in parallel in some embodiments.Additionally, or alternatively, some portions of process 1100 can beomitted in some embodiments. Calibration process 1100 can be implementedin any suitable hardware and/or software. For example, in someembodiments, calibration process 1100 can be implemented in macroinspection system 10. Note, that calibration process 1100 is not limitedto macroscope inspection systems and can also be implemented in amicroscope inspection system.

FIG. 13A shows at a high level, an example method 1300 for illuminatinga specimen using a macro inspection system to achieve a desiredillumination landscape) “illumination landscape method 1300” (inaccordance with some embodiments of the disclosed subject matter. Insome embodiments, illumination landscape method 1300 can use macroinspection system 10.

At 1301, a specimen to be examined can be placed on specimen stage 12.In some embodiments, the specimen is brought into focus before theillumination landscape provided by macro inspection system 10 isadjusted.

At 1302, according to some embodiments, control module 70 can activateand adjust the intensity, color and/or position of lights 16, 50 and/or60, and/or the distance between specimen stage 12 and lens 14 accordingto a stored illumination profile that is selected for the specimen. Theillumination profile can be selected manually or automatically based ona computer algorithm that assesses different attributes of the specimen(e.g., as determined by one or more physical and/or mechanicalproperties of a specimen) and/or different goals of the examination andfinds a suitable illumination profile. Methods for selecting a suitableillumination profile are further discussed in connection with FIGS. 10,13B, and 14.

In some embodiments, after selected lights 16, 50 and/or 60 areactivated at different colors and/or intensity, and the selected lights,and adjustments are made to the intensity, color and/or light position,and/or the distance between specimen stage 12 and lens 14, according toa selected illumination profile, further adjustments can be to modifythe selected illumination profile to achieve a desired illuminationlandscape. In some embodiments, one or more lights 16, 50 and/or 60 canbe activated and adjustments can be made to the intensity, color and/orposition of the lights, and/or the distance between specimen stage 12and lens 14 without reference to any illumination profile. Theactivations and/or adjustments can be performed manually orautomatically.

Once one or more of lights 16, 50 and/or 60 are activated, andadjustments are made to their intensity, color and/or light position, aswell as to the distance between specimen stage 12 and lens 14, one ormore images of the specimen can be captured and stored for analysis, asat 1303. In some embodiments, the captured specimen images aretransmitted to computer analysis system 75.

At 1305, a determination is made by computer analysis system 75 as towhether the applied activation of one or more of lights 16, 50 and/or60, and adjustments to their intensity, color and/or light position,etc. are sufficient to produce a desired illumination landscape. Suchdeterminations may be made based on an analysis of pixel intensityvalues for image data received during the image capture step of 1303. Ifthe illumination landscape profile is determined to be sub-optimal, thenprocess 1300 can revert back to step 1302, and further adjustments tothe illumination landscape can be made. Steps 1302-1305 can iterateuntil an optimal illumination profile is achieved. By way of example, ifan illumination landscape with a uniform light intensity profile isdesired for a particular specimen type, but the image data associatedwith the captured one or more specimen images indicate that some regionsare insufficiently illuminated, then step 1305 can revert back to step1302. In step 1302, additional changes to light activation, intensity,position (elevation and/or pivot/rotation), etc. can be made. Oncechanges have been applied to the illumination landscape, step 1303 isrepeated and image data is collected from the specimen under the newconditions, e.g., by an image capture device. Again, at step 1305, thenew illumination landscape is analyzed to determine if optimal lightingconditions have been achieved.

Different illumination profiles can be selected for a specimen, and foreach selected illumination profile, control module 70 can activate andadjust the intensity, color and/or position of lights 16, 50 and/or 60,and/or distance between specimen stage 12 and lens 14 according to theselected profile, and capture and store one or more images of thespecimen. As such, the iterative process of steps 1302-1305 can differwith specimen type, as the initially applied illumination landscape thatis applied at step 1302 may vary with specimen type, region of interest,a particular stage in the manufacturing or examining process, and/or forany other suitable classification group. In some embodiments, once theillumination is configured according to a selected illumination profile,specimen stage 12 and/or lens 14 can be adjusted to different positionsin relation to each other and one or more images of the specimen can becaptured at each distance.

FIG. 13B illustrates steps of an example process 1310 for identifying aspecimen classification and automatically adjusting an illuminationlandscape of the macro inspection apparatus, according to some aspectsof the disclosed technology. Process 1310 begins with step 1312 whichimage data is received, for example, by an image processing system e.g.,image processing module 1034, discussed above. In some approaches, theimage data can be included in a received image of a specimen that istaken by an imaging device, as part of macro inspection system 10. Theimage data can include all or a portion of a specimen that is disposedon a stage of macro inspection system 10. In some instances, that imagedata may only comprise pixel intensity values, indicating an intensityof light reflected from different portions of a specimen surface.

In step 1314, the image data is analyzed to identify a classification ofthe specimen. In some instances image analysis may be performed toidentify a subset of the specimen, such as a particular region orfeature. As discussed below, machine learning classifiers, computervisions and/or artificial intelligence can be used to identify/classifythe specimen.

Subsequently, in step 1316, an illumination profile can be automaticallyselected based on the specimen (or feature) classification and/or aparticular stage in the manufacturing or examining process. Thespecimen/feature classification can be used to query an illuminationprofile database that contains one or more illumination profilesassociated with specimen and/or specimen feature types. By referencingthe specimen classification determined in step 1314, a matchingillumination profile can be automatically identified and retrieved. Asdiscussed above, the illumination profile can contain a variety ofsettings data that describe configurations of macro inspection system 10that can be used to achieve the optimal illumination landscape for thespecimen or feature being observed. The automatically selectedillumination profile is used to, at step 1318, automatically adjust theillumination landscape.

It should be understood that at least some of the portions ofillumination landscape method 1300 described herein can be performed inany order or sequence not limited to the order and sequence shown in anddescribed in connection with FIGS. 13A and 13B, in some embodiments.Also, some portions of process 1300 described herein can be performedsubstantially simultaneously where appropriate or in parallel in someembodiments. Additionally, or alternatively, some portions of process1300 can be omitted in some embodiments. Illumination landscape method1300 can be implemented in any suitable hardware and/or software. Forexample, in some embodiments, illumination landscape method 1300 can beimplemented in macro inspection system 10. Note, that illuminationlandscape method 1300 is not limited to macroscope inspection systemsand can also be implemented in microscope inspection systems.

FIG. 10 shows the general configuration of an embodiment of computeranalysis system 75, in accordance with some embodiments of the disclosedsubject matter. Although computer analysis system 75 is illustrated as alocalized computing system in which various components are coupled via abus 1005, it is understood that various components and functionalcomputational units (modules) can be implemented as separate physical orvirtual systems. For example, one or more components and/or modules canbe implemented in physically separate and remote devices, such as, usingvirtual processes (e.g., virtual machines or containers) instantiated ina cloud environment.

Computer analysis system 75 includes a processing unit (e.g., CPU/sand/or processor/s) 1010 and bus 1005 that couples various systemcomponents including system memory 1015, such as read only memory (ROM)1020 and random access memory (RAM) 1025, to processor/s 1010.

Memory 1015 can include various memory types with different performancecharacteristics, such as memory cache 1012. Processor 1010 is coupled tostorage device 1030, which is configured to store software andinstructions necessary for implementing one or more functional modulesand/or database systems, such as profile generation module 1032,illumination profile database 1036, and imaging processing module 1034.Each of these modules can be configured to control processor 1010 aswell as a special-purpose processor where software instructions areincorporated into the actual processor design. As such, processor 1010and one or more of profile generation module 1032, illumination profiledatabase 1036, and imaging processing module 1034 can be completelyself-contained systems. For example, imagine processing module 1034 canbe implemented as a discrete image processing system, without departingfrom the scope of the disclosed technology.

To enable user interaction with computer analysis system 75, inputdevice 1045 can represent any number of input mechanisms, such as amicrophone for speech, a touch-sensitive screen for gesture or graphicalinput, keyboard, mouse, motion input and so forth. An output device 1035can also be one or more of a number of output mechanisms known to thoseof skill in the art. In some instances, multimodal systems can enable auser to provide multiple types of input to communicate with computeranalysis system 75, for example, to convey specimen information relatingto a specimen type/classification, or other characteristics.Communications interface 1040 can generally govern and manage the userinput and system output. There is no restriction on operating on anyparticular hardware arrangement and therefore the basic features heremay easily be substituted for improved hardware or firmware arrangementsas they are developed.

Storage device 1030 is a non-transitory memory and can be a hard disk orother types of computer readable media that can store data accessible bya computer, such as magnetic cassettes, flash memory cards, solid statememory devices, digital versatile disks, cartridges, random accessmemories (RAMs) 525, read only memory (ROM) 520, and hybrids thereof.

In practice, illumination profile generation module 1032 can beconfigured to receive a scan of a specimen, or a portion of a specimen(collectively, “specimen image” (from macro inspection system 10, and/orany suitable computer readable media. In some instances, preferredillumination landscapes associated with configurations of the variousmacro components of macro inspection system 10 can be associated to forman illumination profile, for example, that is associated with thespecimen type or classification. Illumination profiles associatingillumination landscape settings with specimen classification types canbe stored to illumination profile database 1036.

Illumination profiles stored to illumination profile database 1036 caninclude specific context data such as: a configuration of lights 16, 50and/or 60 of macro inspection system 10 (e.g., the total number oflights 16, 50 and/or 60, the address and location of each light 16, 50and/or 60, the total number and location of light bars 28, the area ofprojection for each light 16, 50 and/or 60 at each possible positionthat it can be located (including height and angle) from the lightsource to the region where the light is projected); the range ofpossible distances between specimen stage 12 and lens 14; regions ofinterest for particular types of specimen; a particular stage of amanufacturing or examining process that is being examined; a featurethat is being examined.

Image processing module 1034 can be used in conjunction with profilegeneration module 1032 and illumination profile database 1036 toclassify a specimen based on the image data received in the specimenimage(s) and/or other received specimen characteristics, such as thosemanually provided by a user, for example, via input device 1045.Additionally, image processing module can be configured to classifyspecific specimen features, determine other physical and/or mechanicalspecimen properties (e.g., specimen reflectivity, specimen dimensions).Classifications of specimen types, and specimen features/properties canbe stored as part of an illumination profile. As such, variousillumination profiles stored in illumination profile database 1036 cancontain settings and parameters used to generate an optimal illuminationlandscape that can be referenced and matched to a sample based on sampletype and or specific features or characteristics.

In some aspects, classification of a specimen type and/or features of aspecimen can be performed using image processing algorithms that caninclude computer vision, one or more artificial intelligencealgorithm(s) and/or computer algorithms. Classification of a specimen,or features of a specimen, can also be based on, e.g., a computer aideddesign (CAD) file of a specimen and/or features of a specimen, aspecimen layout map identifying features on a specimen, images of knownspecimens and/or features, and/or information about known specimens(e.g., a specimen's dimensions, the mechanical and/or physicalproperties of a specimen).

In some instances, machine learning models can be used to performclassification of specimens, specimen features, and/or other specimencharacteristics. In some aspects, image data from specimen images can beprovided as an input to a machine learning classification system, forexample, by image processing module 1034. Classifier output can specifya sample or feature classification that can then be used to reference anillumination profile stored in illumination profile database 1036. Bymatching the correct illumination profile with the correct sampleclassification or feature type, the correct illumination landscape canbe achieved through the automatic calibration of light intensity, lightcolor, lighting angle, and elevation above the specimen, etc.

As understood by those of skill in the art, machine learning basedclassification techniques can vary depending on the desiredimplementation, without departing from the disclosed technology. Forexample, machine learning classification schemes can utilize one or moreof the following, alone or in combination: hidden Markov models;recurrent neural networks; convolutional neural networks; Bayesiansymbolic methods; general adversarial networks; support vector machines;image registration methods; applicable rule-based system. Whereregression algorithms are used, they may include including but are notlimited to: a Stochastic Gradient Descent Regressor, and/or a PassiveAggressive Regressor, etc.

Machine learning classification models can also be based on clusteringalgorithms (e.g., a Mini-batch K-means clustering algorithm), arecommendation algorithm (e.g., a Miniwise Hashing algorithm, orEuclidean LSH algorithm), and/or an anomaly detection algorithm, such asa Local outlier factor. Additionally, machine learning models can employa dimensionality reduction approach, such as, one or more of: aMini-batch Dictionary Learning algorithm, an Incremental PrincipalComponent Analysis (PCA) algorithm, a Latent Dirichlet Allocationalgorithm, and/or a Mini-batch K-means algorithm, etc.

Such algorithms, networks, machines and systems provide examples ofstructures used with respect to any “means for determining anillumination profile for a specimen using artificial intelligence.”

In some embodiments, machine learning can be deployed in the creation ofillumination profiles. For example, profile generation module 1032 caninput the context data, along with the specimen image or data determinedfrom the specimen image)“specimen data” (into a trained artificialintelligence algorithm to create one or more appropriate illuminationprofiles to be applied to illuminate a specimen. In other embodiments,image processing module 1034 can use machine learning models or othercomputer algorithms to select a predefined illumination profile based onthe specimen image, specimen data and/or context data, as discussedabove.

Once the desired illumination profile has been selected, e.g., fromillumination profile database 1036, the illumination profile data can betransmitted to control module 70. Control module 70 can use thisinformation in connection with process 1300 to apply an illuminationprofile to illuminate a specimen being examined.

Examples of artificial intelligence based image processing algorithmthat can be used by illumination profile generation module 1032 is imageregistration as described by: Barbara Zitova, “Image RegistrationMethods: A Survey,” Image and Vision Computing, Oct. 11, 2003, Volume21, Issue 11, pp. 977-1000, which is hereby incorporated by referenceherein in its entirety. The disclosed methods are just examples and arenot intended to be limiting.

In some embodiments, the machine learning algorithms used byillumination profile generation module 1032, and image processing module1034, including, in some embodiments, an image processing algorithm, isfirst trained with training data so that illumination profile generationmodule 1032 can create an appropriate illumination profile for aspecimen.

As shown in FIG. 14, training data 1401 can include labeled images ofknown specimens and features captured by a macro inspection systemaccording to embodiments of the disclosed subject. The labeled imagesselected for training can be images of desired quality that showsuitable detail based on an inspection objective for the capturedimages. In some embodiments, training data 1401 can include non-imagefiles identifying the type of specimen and/or features being inspected.Training data can further include for each image: data describing theactivation, intensity, color and/or position of lights 16, 50 and/or 60,and/or the distance between specimen stage 12 and lens 14; the featuresof a specimen being inspected; the region of interest on the specimenbeing inspected; the particular stage of a manufacturing or examiningprocess being inspected. In some embodiments training data can includephysical/mechanical properties of a specimen, and/or any other suitablecharacteristic used to create an appropriate illumination profile. Insome embodiments, training data can also include unlabeled data.

Once the artificial intelligence algorithm used by illumination profilegeneration module 1032 is trained, it can be applied by illuminationprofile generation module 1032 to a received specimen scan to create oneor more illumination profiles (output data 1402) for each receivedspecimen image. As described above, illumination profile data caninclude data identifying which lights 16, 50 and/60 to activate, and atwhat intensity, color and light position. Illumination profile data canalso include a distance between specimen stage 12 and lens 14.

Note that automatic macro inspection system 10 can include othersuitable components not shown. Additionally or alternatively, some ofthe components included in automatic mapping macro inspection system 10can be omitted.

In some embodiments, control module 70 of macro inspection system 10 canbe used to locate features of a specimen. Features can refer to knowncharacteristics of a specimen, as well as abnormalities and/or defects.Features can include but are not limited to: circuits, circuit boardcomponents, biological cells, tissue, defects (e.g., scratches, dust,fingerprints). Control module 70, using computer vision techniques(e.g., artificial intelligence based algorithms, as discussed above, orother algorithms for image processing or pattern recognition), or otherknown techniques for image analysis, can detect features on a specimen.For each feature it detects, control module 70 using computer vision orother known image analyses techniques can identify a feature's centroid.As described below, control module 70, can apply different coordinatesystems to define the X, Y location of a feature's centroid, as well asthe orientation of a feature on a specimen.

FIG. 15 illustrates a rectangular shaped specimen S₁ and an examplecoordinate system used to locate one or more features (e.g.,representative features F₁, F₂). In some embodiments, control module 70using computer vision techniques can identify the centroid of eachfeature F₁, F₂ on S₁. As shown in FIG. 15, a Cartesian XY coordinatesystem can be used to define the X, Y coordinate location of eachcentroid F1 (e.g., x₁, y₁), F2 (e.g., x₂, y₂). The XY coordinatelocation of each centroid represents a distance from coordinate axes X,Y that meet at origin point (O). In some embodiments, the coordinateaxes can be a pair of perpendicular lines that extend from a corner ofspecimen S 1. Note that coordinate axes X and Y and origin point O arejust examples, the coordinate location of a feature can be measured fromother coordinate axes and origin point O and/or from another referencepoint(s). In other embodiments, a feature can be located by: its polarcoordinates in relation to an origin point and/or any other suitablelocation. Similar methods can be used to find features on differentlyshaped specimens.

In some embodiments, as shown for example in FIGS. 16A and 16B, thecoordinate axes X, Y used to locate a feature on a specimen can be basedon two reference indices (e.g., R₃ and R₄). For example, a firstcoordinate axis, as shown in FIG. 16B, can extend through the tworeference indices R₃ and R₄. A second coordinate axis can be positionedperpendicular to the first coordinate axis, so that the intersection ofthe two axes forms an origin point (O). The second coordinate axis canintersect the first coordinate axis at one of the reference indices R₃and R₄, or it can also be positioned anywhere along the first coordinateaxis. Specifically, FIG. 16B illustrates an irregularly shaped specimenS₃ and example coordinate axes X and Y extending from R₃ and R₄ used tolocate the centroid of one or more features (e.g., representativefeature F₄). Note, R₃ and R₄ can be intentionally positioned on aspecimen or can be naturally occurring marks on the specimen. In someembodiments, control module 70 can use computer vision algorithms todetermine reference marks R₃ and R₄ that can be used to establish theperpendicular coordinate axes and to locate features on a specimen.

In some embodiments, control module 70 can use computer visiontechniques to identify a feature's orientation relative to a pair ofcoordinate axes. In FIG. 17, the angle of rotation for feature F₁ of S₁is shown relative to the coordinate X axis and identified by the angleα. This angle α can be calculated by known means of computer vision,and, in such embodiments the x₁, y₁ coordinates and angle α can uniquelylocate the position and orientation of a feature such as feature F₁ on aspecimen. Although shown with respect to a rectangular specimen S₁, itwill be appreciated that the calculation of an angle α can be practicedwith respect to any shaped specimen.

In some embodiments, a specimen to be inspected can be registered on thespecimen stage of macro inspection system 10 in a specific orientationusing an indicator on a specimen (e.g., a notch, flat) and aligning theindicator with respect to the known coordinate axes of the specimenstage to locate features on the specimen. In other embodiments,particularly when employing reference marks on the specimen, thespecimen can be placed on the specimen stage in any orientation, and thereference marks on the specimen can be employed to establish thecoordinate axis, as discussed in connection with FIG. 17.

Note, the methods described herein to locate features of a specimen isnot limited to macroscope inspection systems and can also be implementedin microscope inspection systems.

In some embodiments, any suitable computer readable media can be usedfor storing instructions for performing the functions and/or processesdescribed herein. For example, in some embodiments, computer readablemedia can be transitory or non-transitory. For example, non-transitorycomputer readable media can include media such as non-transitorymagnetic media (such as hard disks, floppy disks, etc.), non-transitoryoptical media (such as compact discs, digital video discs, Blu-raydiscs, etc.), non-transitory semiconductor media (such as flash memory,electrically programmable read only memory (EPROM), electricallyerasable programmable read only memory (EEPROM), etc.), any suitablemedia that is not fleeting or devoid of any semblance of permanenceduring transmission, and/or any suitable tangible media. As anotherexample, transitory computer readable media can include signals onnetworks, in wires, conductors, optical fibers, circuits, and anysuitable media that is fleeting and devoid of any semblance ofpermanence during transmission, and/or any suitable intangible media.

The various systems, methods, and computer readable mediums describedherein can be implemented as part of a cloud network environment. Asused in this paper, a cloud-based computing system is a system thatprovides virtualized computing resources, software and/or information toclient devices. The computing resources, software and/or information canbe virtualized by maintaining centralized services and resources thatthe edge devices can access over a communication interface, such as anetwork. The cloud can provide various cloud computing services viacloud elements, such as software as a service (SaaS) (e.g.,collaboration services, email services, enterprise resource planningservices, content services, communication services, etc.),infrastructure as a service (IaaS) (e.g., security services, networkingservices, systems management services, etc.), platform as a service(PaaS) (e.g., web services, streaming services, application developmentservices, etc.), and other types of services such as desktop as aservice (DaaS), information technology management as a service (ITaaS),managed software as a service (MSaaS), mobile backend as a service(MBaaS), etc.

The provision of the examples described herein (as well as clausesphrased as “such as,” “e.g.,” “including,” and the like) should not beinterpreted as limiting the claimed subject matter to the specificexamples; rather, the examples are intended to illustrate only some ofmany possible aspects. A person of ordinary skill in the art wouldunderstand that the term mechanism can encompass hardware, software,firmware, or any suitable combination thereof.

Unless specifically stated otherwise as apparent from the abovediscussion, it is appreciated that throughout the description,discussions utilizing terms such as “determining,” “providing,”“identifying,” “comparing” or the like, refer to the action andprocesses of a computer system, or similar electronic computing device,that manipulates and transforms data represented as physical(electronic) quantities within the computer system memories or registersor other such information storage, transmission or display devices.Certain aspects of the present disclosure include process steps andinstructions described herein in the form of an algorithm. It should benoted that the process steps and instructions of the present disclosurecould be embodied in software, firmware or hardware, and when embodiedin software, could be downloaded to reside on and be operated fromdifferent platforms used by real time network operating systems.

The present disclosure also relates to an apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, or it may comprise a general-purpose computerselectively activated or reconfigured by a computer program stored on acomputer readable medium that can be accessed by the computer. Such acomputer program may be stored in a computer readable storage medium,such as, but is not limited to, any type of disk including floppy disks,optical disks, CD-ROMs, magnetic-optical disks, read-only memories(ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic oroptical cards, application specific integrated circuits (ASICs), or anytype of non-transient computer-readable storage medium suitable forstoring electronic instructions. Furthermore, the computers referred toin the specification may include a single processor or may bearchitectures employing multiple processor designs for increasedcomputing capability.

The algorithms and operations presented herein are not inherentlyrelated to any particular computer or other apparatus. Variousgeneral-purpose systems may also be used with programs in accordancewith the teachings herein, or it may prove convenient to construct morespecialized apparatus to perform the required method steps andsystem-related actions. The required structure for a variety of thesesystems will be apparent to those of skill in the art, along withequivalent variations. In addition, the present disclosure is notdescribed with reference to any particular programming language. It isappreciated that a variety of programming languages may be used toimplement the teachings of the present disclosure as described herein,and any references to specific languages are provided for disclosure ofenablement and best mode of the present disclosure.

The macro inspection mechanism, method and system have been described indetail with specific reference to these illustrated embodiments. It willbe apparent, however, that various modifications and changes can be madewithin the spirit and scope of the disclosure as described in theforegoing specification, and such modifications and changes are to beconsidered equivalents and part of this disclosure. The scope of thepresent disclosure is limited only by the claims that follow.

The invention claimed is:
 1. An inspection apparatus comprising: a stageconfigured to retain a specimen for inspection; an imaging device havinga field of view encompassing at least a portion of the stage to viewlight reflected by the specimen retained on the stage; a plurality oflights disposed on a moveable platform; and a computing system coupledto the imaging device, each of the plurality of lights, and the moveableplatform, wherein the computing system is configured to performoperations comprising: generating a trained prediction model to generatean illumination profile for the specimen based on a training data setcomprising a plurality of known specimens and associated featurescaptured by the imaging device; generating, via the trained predictionmodel, a plurality of desired illumination profiles for a plurality ofreference specimens; receiving image data from the imaging device,wherein the image data indicates one or more physical and mechanicalproperties of a target specimen; selecting a first illumination profilebased on the one or more physical and mechanical properties of thetarget specimen; analyzing the image data to determine whether a targetillumination profile of the target specimen is within a threshold valueof the first illumination profile; and automatically modifying anelevation of the moveable platform or an intensity of one or more of theplurality of lights to achieve the first illumination profile, based onthe analyzing.
 2. The inspection apparatus of claim 1, wherein thetraining data set further comprises non-image files identifying a typeof specimen and/or associated feature being inspected.
 3. The inspectionapparatus of claim 1, wherein, for each image in the training data setcomprises one or more of data describing an activation, intensity,color, position of the plurality of lights, or a distance between thestage and lens of the imaging device.
 4. The inspection apparatus ofclaim 1, wherein the operations further comprise: automaticallyadjusting a color of one or more of the plurality of lights according tothe first illumination profile.
 5. The inspection apparatus of claim 1,wherein the operations further comprise: automatically adjusting a pivotof one or more of the plurality of lights according to the firstillumination profile.
 6. The inspection apparatus of claim 1, whereinthe operations further comprise: receiving specimen data indicating aspecimen classification associated with the specimen, and whereinautomatically modifying the elevation of the moveable platform or theintensity of one or more of the plurality of lights is further based onthe specimen data.
 7. The inspection apparatus of claim 1, wherein theoperations further comprise: receiving specimen data indicating one ormore physical properties associated with the specimen, and whereinautomatically modifying the elevation of the moveable platform or theintensity of one or more of the plurality of lights is further based onthe specimen data.
 8. A method, comprising: generating, by a computingsystem coupled to an imaging device of an inspection apparatus, atrained prediction model to generate an illumination profile for aspecimen positioned on a stage of the inspection apparatus based on atraining data set comprising a plurality of known specimens andassociated features captured by the imaging device; generating, by thecomputing system via the trained prediction model, a plurality ofdesired illumination profiles for a plurality of reference specimens;receiving, by the computing system, image data from the imaging device,wherein the image data indicates one or more physical and mechanicalproperties of a target specimen; selecting, by the computing system, afirst illumination profile based on the one or more physical andmechanical properties of the target specimen; analyzing, by thecomputing system, the image data to determine whether a targetillumination profile of the target specimen is within a threshold valueof the first illumination profile, and automatically modifying, by thecomputing system, an intensity of one or more of a plurality of lightsdisposed on a moveable platform of the inspection apparatus or anelevation of the moveable platform to achieve the first illuminationprofile, based on the analyzing.
 9. The method of claim 8, wherein thetraining data set further comprises non-image files identifying a typeof specimen and/or associated feature being inspected.
 10. The method ofclaim 8, wherein, for each image in the training data set comprises oneor more of data describing an activation, intensity, color, position ofthe plurality of lights, or a distance between the stage and lens of theimaging device.
 11. The method of claim 8, further comprising:automatically adjusting, by the computing system, a color of one or moreof the plurality of lights according to the first illumination profile.12. The method of claim 8, further comprising: automatically adjusting,by the computing system, a pivot of one or more of the plurality oflights according to the first illumination profile.
 13. The method ofclaim 8, further comprising: receiving, by the computing system,specimen data indicating a specimen classification associated with thespecimen, and wherein automatically modifying the elevation of themoveable platform or the intensity of one or more of the plurality oflights is further based on the specimen data.
 14. The method of claim 8,further comprising: receiving, by the computing system, specimen dataindicating one or more physical properties associated with the specimen,and wherein automatically modifying the elevation of the moveableplatform or the intensity of one or more of the plurality of lights isfurther based on the specimen data.
 15. A non-transitorycomputer-readable storage medium storing thereon executable instructionswhich, when executed by one or more processors of a computing systemcoupled to an imaging device of an inspection apparatus, cause thecomputing system to perform operations, comprising: generating, by thecomputing system, a trained prediction model to generate an illuminationprofile for a specimen positioned on a stage of the inspection apparatusbased on a training data set comprising a plurality of known specimensand associated features captured by the imaging device; generating, bythe computing system via the trained prediction model, a plurality ofdesired illumination profiles for a plurality of reference specimens;receiving, by the computing system, image data from the imaging device,wherein the image data indicates one or more physical and mechanicalproperties of a target specimen; selecting, by the computing system, afirst illumination profile based on the one or more physical andmechanical properties of the target specimen; analyzing, by thecomputing system, the image data to determine whether a targetillumination profile of the target specimen is within a threshold valueof the first illumination profile; and automatically modifying, by thecomputing system, an intensity of one or more of a plurality of lightsdisposed on a moveable platform of the inspection apparatus or anelevation of the moveable platform to achieve the first illuminationprofile, based on the analyzing.
 16. The non-transitorycomputer-readable storage medium of claim 15, wherein the training dataset further comprises non-image files identifying a type of specimenand/or associated feature being inspected.
 17. The non-transitorycomputer-readable storage medium of claim 15, wherein, for each image inthe training data set comprises one or more of data describing anactivation, intensity, color, position of the plurality of lights, or adistance between the stage and lens of the imaging device.
 18. Thenon-transitory computer-readable storage medium of claim 15, furthercomprising: automatically adjusting, by the computing system, a color ofone or more of the plurality of lights according to the firstillumination profile; or automatically adjusting, by the computingsystem, a pivot of one or more of the plurality of lights according tothe first illumination profile.
 19. The non-transitory computer-readablestorage medium of claim 15, further comprising: receiving, by thecomputing system, specimen data indicating a specimen classificationassociated with the specimen, and wherein automatically modifying theelevation of the moveable platform or the intensity of one or more ofthe plurality of lights is further based on the specimen data.
 20. Thenon-transitory computer-readable storage medium of claim 15, furthercomprising: receiving, by the computing system, specimen data indicatingone or more physical properties associated with the specimen, andwherein automatically modifying the elevation of the moveable platformor the intensity of one or more of the plurality of lights is furtherbased on the specimen data.